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ABSTRACT 

Although the Juvenile Justice and Delinquency 
Prevention Act of 1974 does not specify the primary purpose of the 
Special Emphasis Program, it is generally categorized as serving a 
dual function of research/demonstration and service delivery. A 
General Afccounting Office (GAO) review was held to determine the 
efficiency and effectiveness of the management of the program. The 
results indicated that the Office of Juvenile Justice and Delinquency 
Prevention did not recognize the need to operate the two programs in 
a different manner. GAO found that the office's policies and 
procedures primarily addressed service delivery and virtually ignored 
research/demonstratio*A In response to these findings, the Department 
of Justice, although disagreeing on some points, stated that 
corrective action would be taken. (JAC) 



**************************************************************** 

* Reproductions supplied by EDRS are the best that can be made * 

*, from the original document. * 

*********************************************************************** 



ERLC 



1 



BY THE U.S. GENERAL ACCOUNTING OFFICE 
£ Report To The Attorney General 



The Office Of Juvenile Justice And Delin 
quency Prevention's Special Emphasis Pro 
gram Has Not Realized! Its Full Potential 



Th€ Office of Juvenile Justice and Delinquen- 
cy Prevention needs to establish policies and 
procedures for its Special Emphasis Program 
which recognize that research/demonstration 
and service delivery initiatives should be de- 
signed and managed differently. GAO con- 
cluded that unless this was done, the Office 
would achieve only limited results from its 
research/demonstration progcams. 

The Department of Justice, although dis- 
agreeing on some points, concurred with 
GAO's recommendations and stated that 
corrective action would be taken. 



William J. Anderson, Director 
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Request for copies of GAO reports should be 
sent to: 

U.S. General Accounting Office 
Document Handling and Information 

Services Facility 
P.O. Box 6015 
Geithersburg, Md. 20760 



Telephone (202) 275-6241 

The first five copies of individual reports are 
free of charge. Additional copies of bound 
$ audit reports are $3.25 each. Additional 

copies of unbound report (i.e., letter reports) 
and most other publications are $1.00 each. 
There will be a 25% discount on all orders for 
100 or more copies mailed to a single address. 
Sales orders must be prepaid on a cash, check, 
or money order basis, Check should be made 
out to the "Superintendent of Documents". 
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United states general accounting office 

WASHINGTON, D.C. 20548 



ummi uovowmikt 
oivisioh 



.B-202245 



The Honorable William French Smith 
The Attorney General 

Dear Mr. Attorney General: 



This report discusses the need for the Office of Juvenile 
Justice and Delinquency Prevention to establish policies and 
procedures which recognize that its special emphasis research/ 
demonstration and service delivery programs should be designed 
and managed differently. 

The report makes recommendations to you on pages 23 and 
24 As you know, Section 236 of the Legislative Reorganiza- 
tion Act of 1970 requires the head of a Federal agency to sub- 
rait a written statement on actions taken on our recommendations 
to the House Committee on Government Operations and the Senate 
Committee on Governmental Affairs not later than 60 days after 
the date of the report and to the House and Senate Committees on 
Appropriations with the agency's first request for appropriations 
made more than 60 days after the date of the report. 

We are sending copies of this report to the Director, 
Office of Management and Budget. 

Sincerely yours, 

William J. Anderson 
Director 



/ 
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GENERAL ACCOUNTING OFFICE 
REPORT *0 THE ATTORNEY GENERAL 



THE OFFICE OF JUVENILE JUSTICE 
AND DELINQUENCY PREVENTION'S 
SPECIAL EMPHASIS PROGRAM HAS 
NOT REALIZED ITS FULL POTENTIAL 



DIGEST 

The Office of Juvenile Justice- and Delinquency 
Prevention, the agency responsible for adminis- 
tering the Special Emphasis Program, needs to 
establish policies and procedures addressing 
the Program's different mandates. 

From fiscal year 1975 through 1981, about $223 
million was made available to fund the Special 
Emphasis Prevention and Treatment Program. 
Under this program, which is authorized by the 
juvenile Justice and Delinquency Prevention 
Act of 1974, grants and contracts can be awarded 
for various* objectives including 

—developing and implementing new approaches, 
techniques, and methods with respect to 
juvenile delinquency programs; 

—improving the capability of public and Private 
agencies and organizations to provide services 
for delinquents and other youth to help prevent 
delinquency; and 

—developing and implementing prevention and 
-treatment programs relating to juveniles who 
commit serious crimes. 

Although the act does not specify the Program's 
Ittlmary purpose, it can generally be categorized 
as serving the dual function of research/demon- 
stration and service delivery. GAO's review, 
which was made to determine how efficiently and 
effectively the Office of Juvenile Justice and 
Delinquency Prevention managed the Special 
Emphasis Program, showed that the Office did not 
recognize the need to operate these types of pro- 
grams differently. GAO found that the Office s 
policies and procedures primarily addressed serv- 
ice delivery and virtually ignored research/ 
demonstration. (See pp. 1 to 10). 

Various studies indicate that several critical 
elements are involved in succesafully implement- 
ing a research/demonstration effort. ~heae 
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elements include: (1) conductinq research on a 
national scope issue; (2) developinq a proqram 
on the basis of research findings; (3) testing 
and evaluating the program rigorously; and 
(4) demonstrating it in several jurisdictions 
with varying characteristics. Service delivery 
programs , on the other hand, do not need the 
same degree of testing and evaluation because 
their purpose is to address a localized need 
and not to show that the effort is effective 
and can be implemented nationwide on a cost 
effective basis* (See p. 9.) 

GAO reviewed three research/demonstration 
initiatives totaling $31 million in grant or 
contract awards and found problems with the 
design and management of two of them. Specifi- 
cally: 

— Program announcements failed to clearly 
establish research/demonstration as the 
initiatives' purpose. (See pp. 11 to 13.) 

— Research was not adequately incorporated 

into the initiatives 1 development and design . 
(See pp . 14 to 17 . ) 

— Evaluation was not made an integral part of , 
the initiatives. (See pp. 17 to 21.) 

— Technical assistance was not always delivered 
to grantees. (See pp. 21 and 22.) 

— The amount and type of monitoring required 
for research/demonatrat ion programs were not 
determined. (See p. 22.) 

The third initiative GAO reviewed — Violent 
Juvenile Offender — had a design which was more 
compatible with research/demonstration condi- 
tions . However, the Office cannot ensure that 
this will consistently occur unless it rec- 
ognizes in its policies and procedures that 
research/demonstration and service delivery pro- 
grama have different objectives and need to be 
operated differently. Developing such procedures 
takes on added a ignif icance in light of the re- 
port of the Attorney General's Task Force on 
Violent Crime. The Task Force's report recom- 
mends that the Attorney Genera 1 ensure that 
adequate resources are available for research, 
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development, demonstration, and independent 
evaluation of methods to prevent and reduce 
serious crime. (See pp. 9, 16, and 20.) 

RECOMMENDATIONS TO 
THE ATTORNEY GENERAL 

GAG recommends that the Attorney General di- 
rect the Administrator of the Office of Juvenile 
Justice and Delinquency Prevention to establish 
policies and procedures which recognize that 
research/demonstration and service delivery 
programs should be designed and managed dif- 
ferently. With respect to research/demonstra- 
tion initiatives, the Administrator should 
ensure that 

— program announcements clearly establish /the 
initiatives * purpose; 

— research/demonstration initiatives are ^ 
developed on the basis of research results; 

— evaluation is made an inteqral part of 
research/demonstration initiatives ; 

— technical assistance is available to grantees, 
especially during key phases. 

The Administrator should also determine whether 
research/demonstration programs should be moni- 
tored the same as service delivery proqrams or 
given closer scrutiny. 

AGENCY COMMENTS 

The Department of Justice commented on a draft 
of this report by letter dated March 3, 1982. 
( S«e app . ) 

The Department agreed with all of GAO's recom- 
mendations and noted that the Office of Juvenile 
Justice and Delinquency Prevention would develop 
policies and procedures for each^type of pro- 
gramming possible under the Special Emphasis Pro- 
gram. However, it disagreed with GAO' s conclu- 
sion that research and demonstration proqrams had 
not been adequately designed and managed by 
pointing out that two of the three initiatives 
GAO reviewed were not intended to be research/ 
demonstration programs . 
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After reexamininq available documentation on 
the two initiatives, GAO continues to believe 
that its position that the two initiatives were 
research/demonstration-oriented is well-founded . 
But regardless of what the initiatives were 
supposed to be, it is clear that there was not 
a common understanding among all participants 
of the purposes to be achieved. The corrective 
action promised should help to avoid confusion 
in the future regarding the purpose of the 
various special emphasis initiatives that are 
funded under the program. (See pp. 24 and 25.) 
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INTRODUCTION 



Th. Juvenile uu.tic. ^J^S^^S^^^* 
(42 U.S.C. 5601 et seg.) was enacted to pro i 
resources, leadeTshlnt and coordination to 

-dev.lop and implement effective methods of preventing 
and reducing juvenile delinquency; 

-develop and conduct effective programs to P"™^ 

alternatives to institutionalization; 
-improve the quality of juvenile justice in the United 
States; and 

v ~# Qf .fa and local governments and 

— increase the capacity of State effective juvenile 

prevention. 

.v.nUon W-£> "i^l" th. Department of Ju.tice. 

Th. .ct euthori.ee M»>1 "? l :"" C l!l"™h..t."Sra!.?ror 
program, through formul, . gr ant, , an ^^.^"tat.. (and 
contract.. Formula grants, w "ion relative popula- 

territorie.) generally on the ba.i. ot t >nfl ^ be 

tlon under age 18. are admini.tered by ™ 8tabll>hiBq , OD eratin<,. 
used either to ...i.t them in provide .ubqrants 

tives specified in the legislation, including 

, 4 i. m „nMna new approaches, techniques, 

-2rssa.ri«J , ?isst i ss s.-5. p^- 

grans; 

-developing and maintaining community-based alternatives 
" to traditional forms' of institutionali.atxon , 
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--developing and implementing effective means of divertinq 
juveniles from the traditional juvenile justice and 
correctional system; 

— improving the capability of public and private agencies 
and organizations to provide services for delinquents 
and other youth to help prevent delinquency; 

—developing statewide programs through the use of subsidies 
or other financial incentives designed to remove juveniles 
trom jails and lockups for adults, replicate programs s 
designated as exemplary, or establish and .adopt standards' 
ror the improvement of juvenile justice within the States; 

— improving the juvenile justice system to conform to 
standards of due process; and 

—developing and implementing prevention and treatment 
programs relating to juveniles who commit serious 
crimes . 

SPECIAL EMPHASIS PROGRAMS 
" GENERALLY INVOLVE EITHER 
RZSEARCH/DEMONSTRATION~OR 
SERVICE DELIVERY \ 

Although the act is not specific on the Special Emphasis 
Program s primary purpose, it can generally be categorized as 
serving the dual function of research/demonstration and service 
delivery. For example, the first objective listed in the act 
is to develop and implement new approaches, techniques, and 
methods with respect to juvenile delinquency programs." This 
suggests a research/demonstration purpose in that it aims to 
create new knowledge and methodology. Another objective of 
the program ia fco "improve the capability of public and private 
agencies and organizations to provide services for delinquents 
and other youths to help prevent delinquency." Although the 
statement suggests research/demonstration in that funds might 
be used to show how agencies could more effectively provide 
services, it could also be interpreted as providing funds to 
increase services — a service delivery purpose. 

The act's provisions relating to funding considerations 
also recognize both purposes. Among other thinqs, the act 
states that the Administrator of OJJDP should consider "the 
extent to which the proposed program will incorporate new or 
innovative techniques," as well as "the increase in capacity of 
the public and private agency, institution, or individual to 
provide services to delinquents and other youth to help prPvont 
delinquency. , 
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Funding of program 

From fiscal year 1975 through fiscal year 1^81, about 
$223 million has been made avtilable for apecial emphasis P r 
vention and treatment programs. The money ** mn " aBd mm t ° 
fund various programs or initiatives that usually address an 
issue of national scope. The following chart provides more 
detailed information. 

it 
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Initiative 



De institutional ization 
of status offenders 

Diversion 

School crime 

Prevention- youth skills 
development 

Restitution 

Children in custody 

Prevention research afnd 
development 



Removal of juveniles 
from adult jails and 
lockups 

New pride replication 

Violent juvenile 
offender , 

Alternative educa- 
tion (note b) 

Youth advocacy 

Others (note c) 

Total . 



Amount 
awarded 



$13, 835", 261 
$11,830,605 
$ 9 ,412,5*7 

$16,601,680 
$21,663 ,189 
$ 6,640,053 

$ 2,267,906 

V 

$ 5 ,558 ,000 
$ 9,249,172" 

$ 4,311,998 

$10 ,944 ,352 
$13,945,936 
$51 ,7^3 ,161 
$ 177 ,993 ,550 



Number of 
g r ants/con tracts 
Oiote a ) 

22 
35 
4 

45 

52 
6 

8 

1 
12 



17 
22 
190 
416 



a/This figure includes refunding or continuations. 

"b/The amount awarded fdr the Alternative Education initiative 
includes $3,000,000 contributed by the Department of 
Labor's Office of Youth Programs. 

c/This category v includes a number of initiatives which were 
not targeted f^rmajor emphasis by OJJDP, such as Special 
Emphasis Capacity Building ,' Model Programs, Track II, 
Unsolici ted •( awards) , and Miscellaneous Categories. , These 
initiatives funded State and local organizations for the 
general purpose of preventing delinquency, developing and 
using alternatives to the juvenile justice system, and improv 
ing j uvenile, j ustice administration. 
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The National Institute ,o£ Juvenile [^^jfi™^™* 
Prevention is the organization P^J^J 1C for the evalua- 

its functions, the responsibility of pcovioing etermine 
tion of ^venile delinquency programs ^° d t ute awarded grants 
their results and ef fective *J 0 S;„ fr0 m fiscal years 
or contracts totaling $12 ;J ""J 0 " h f initiatives except Pe- 
1975 through 1981 to evaluate ^ X ^ e f f^ups , Children in 
m oval of juveniles ^-^^hJ^MSgo??. Agency officials 
Custody, and those in the "jj ^ally mentioned above 
told us that the two initiatives s P ec "J c "J i id red t o be suit- 
Sere not ev-luated because they wer «>* h^-SSers" cate- 
able for evaluation. ^ init « 1 ^ t eva i ua ted . 
gory had a limited scope and similarly were 

OBJECTIVES f SCOPE. AN D METHODOLOGY 

we made our review to determine how effectively and effi- 
cient^ OJJDP managed the Special Emphasis Program. 

At OJJDP headquarters - f ^Uy ex amined J.cord. . - 
° f 811 ""J SSiS. P SS tL bas s of e the problem8 y identified, 

^^^^^^B^ f - these three ■ 

•initiatives amounted to about $31 million. 

tors, and subcontractors were located in these areas. 

Each of the three initiatives we reviewed had been devel- 

-S^n ^icp "pcog « ana prices .J^*^-;: 

e°b?ua P r h n US "s iirst year o£ Implementation , and the 

ha ^ f wa 5l^„? ?n. initiati«". tw components. Another reason 
& Seeing tne Violent Senile Offender initiative jm that 
MJDP ^sed contractors, termed National Prooram Coordinators, 



to manage all aspects of the project includinq program develop- 
ment and design/ proposal review, selection of subcontractors, 
and programmatic and fiscal monitoring of the orojects. These 
functions were performed by OJJDP in previous initiatives. 

The criteria for analyzing- the initiatives were obtained 
from the Juvenile Justice and Delinquency Prevention Act and 
related legislative history, oversight hearinqs, regulations, 
the Program's Operation Policy and Procedures Manuai, and 
agency studies* We also obtained additional criteria from 
selected studies obtained through a literature search. 

At OJJDP in Washington, D.C., we interviewed numerous offi- 
cials, including the Acting Administrator of OJJDP, the Director 
of the Special Emphasis Division, the Director of the Formula 
Grants and Technical Assistance Division, the Director of the 
National Institute of Juvenile Justice and Delinquency Preven- 
tion, and the Program Managers for the Prevention, Alternative 
Education, and Violerft Juvenile Offender initiatives. We also 
reviewed the program and evaluation files' for the three 
initiatives. 

To better understand the role that-private contractors had 
in planning and developing special emphasis initiatives, we 
interviewed officials and reviewed files from the American In- 
stitute for Research and from several assessment centers that 
are funded by the National Institute of Juvenile Justice and " 
Delinquency Prevention. The American Institute for Research 
had been involved in the planning of special emphasis initia- 
tives during the Program's early yeairs. The assessment centers 
were responsible for gathering, synthesizing, and assessing 
information on. specific juvenile delinquency issues. We 
visited all four assessment centers: The Assessment Center for 
Integrated Data Analysis; the Assessment Center for Delinquent 
Behavior and Prevention; the Assessment Center for Juvenile 
Justice Systems; and the Assessment Center for Alternatives to 
Juvenile Justice System Processing. 

To obtain information on how well OJJDP managed the 
initiatives, the evaluations, and the technical assistance 
provided, we visited and interviewed representatives of 7 of 
the 15 Prevention initiative grantees and 6 of tfie 17 Alterna- 
tive Education initiative grantees. The grantees were selected 
on the basis of^the high levels of special emphasis funding 
that the States in which they were located received. 

We also visited and interviewed representatives of 

— the National Program Coordinators for the Violent 
Juvenile Offender initiative — thk National Center for 
Crime and Delinquency and the L. Miranda Corporation; 



— the evaluator of the Prevention initiative— the National 
Center for Crime and Delinquency; 

— the evaluator for the Alternative Education initiative 
^ and -its subcontractor— the Johns Hopkins University s 

Center for Social Organization of Schools and the Social 
Action Research Center, respectively; 

— the evaluator for the Violent Juvenile Offender initia- 
. i:: " tive — the ORSA institute; and 

— the technical assistance contractor for the Prevention 
and Alternative Education initiatives— Westinghouse 
National Issues Center. 

Further, we interviewed by telephone the technical assistance 
contractors for the Violent Juvenile Offender initiative— the 
National Office for Social Responsibility and .the Center for 
Community Change. 

The review was performed in accordance with,GAO's current 
-Standards For Audit of Governmental Organizations, Programs, 
Activities, and Functions." 
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CHAPTER 2 



UNLESS OJJDP IMPROVES ITS POLICIES AMD 
PROCEDURES RESEARCH/DEMONSTRATION 
INITIA TIVES W ILL ACHIEVE ONLY LIMITED RESU LTS 

. ' S : — 

Different studies have noted that research/ demonstrations 
in the social arena are diF?5Tcult to successfully implement and 
must be rigorously designed and managed. 1/ OJJDP 1 s policies 
and procedures for the Special Emphasis" Program do not conform 
to the design and management principles in that they primarily 
address service delivery and virtually ignore research/ 
demonstration. As a result: 

— Program announcements die} not clearly establish the 
purpose of initiatives that were intended to be 
research/ demonstration. 

— Research was not adequately incorporated into the 
initiatives' development and design. 

— Evaluation was not made an integral part of the initia- 
tives . 

— Technical assistance was not always provided to 
grantees . 

— The amount and type of monitoring required for 

research/demonstration programs were not determined. 

We examined three research/demonstration initiatives 
that the OJJDP has conducted under its Special Emphasis Pro- 
gram and found that two will produce only limited results. 
The third initiative^ may achieve better results since it had a 
design which was more compatible with research/demonstration 
„ conditions. However, OJJDP cannot ensure tTiat the improvements 
noted in the third initiative will be consistently applied if 
it does not recognize in its policies and procedures that 
research/demonstratipn and service delivery programs need to be 
operated differently. 
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l/Walter S. Baer, Leland L. Johnson, Edward W. Marrow, 
Analysis of Federally Funded Demonstration Projects : 
Final Report" , R-1926-DOC (Santa Monica, Ca . : Rand Cor- 
porat ion ,<> 1976 ) ; The Role of Demonstrations in Federal 
R&D Policy (Washington, D.C: Office of Technology 
Assessment, 197B) ; Studies in the Management of Social 
R h k D; Selected Policy Areas (Washington , 0. C . : 
National Academy of Sciences, 1979). 



1.7 



Developing management policies and procedures to ensure 
that research/demonstration initiatives are effectively designed 
and managed takes on added significance in light of the report 
of the Attorney General's Task Force on Violent Crime. The Task 
Force's report recommends that the Attorney General ensure that 
adeouate resources are available for research, development, 
demonstratrin, and independent evaluation of methods to prevent 
and reduce serious crime. 

(VT .T.,P'S POLICIES AND PROCEDURES DO NOT 
RECOGNIZ E THE NEED TO OPERAT E RESEARCH/ 
DEMONSTRATION AND SERVICE DEL IVERY PROGRAMS 
DIFFERENTLY 

Several critical elements are involved in the successful 
implementation of a research/demonstration effort. In the case 
of OJJDP, it should conduct research on a national scope issue; 
develop programs on the basis of research findings; test and 
evaluate the program' rigorously; and demonstrate it in several 
jurisdictions with varying characteristics to ensure successful 
implementation in different jurisdictions. Ideally, before a 
program can be implemented in different jurisdictions, it 
should be demonstrated and proven effective by an independent 
and rigorous evaluation. Service delivery programs, on the 
other hand, do not need the same type of testing and evaluation 
because their purpose is not to show that the effort can be 
implemented successfully nationwide on a cost effective basis. 
OJJDP needs to recognize; the differences between the purposes— 
research/demonstration and service delivery— of the Special 
Emphasis Program. 

OJJDP' s Operation Policy and Procedures Manual does 
not address the need to operate research/demonstration and 
service delivery programs differently. In fact, the Policies 
and procedures contained in the manual address primarily the 
Special Emphasis Program's service delivery purpose and vir- 
tually ignore research/demonstration. 

The Special Emphasis Program's mission statement contained 
in the manual recognizes that research/demonstration is one of 
the program's purposes by stating that the Special Emphasis 
Division's responsibility is to test, demonstrate, and implement 
program strategies. However, the manual does not provide any 
guidance on how such programs should be designed and operated. 
One unintended result of this omission is that the manual, by 
prescribing conditions that are generally suitable for service 
delivery, has highlighted elements that are not suitable for a 
research/demonstration activity. ''For example, the manual states 
that : 



— Each program initiative has focused upon delivery 
system(s) in relatibn to specific categories of 



\ 

s 

.J 

juveniles to enhance accountability and measurability 
of program activities/services./ 

— Sizeable grants have been awarded for multiyear funding 
to Support program continuity, to demonstrate Federal 
commitment to program approaches, and to underscore the 
scope/trf the problem thus increasing national and local 
attention to the problem addressed. 

— <3oals and objectives for implementation have been 
precisely defined to ensure focus and increase 
accountability, but program design has been left to 
applicants to ensure that projects would respond to 
local conditions . 

OJJDP has funded initiatives with sizeable grants, numer- 
ous sites, and locally developed program designs — conditions 
that various studies do not recommend for research/demonstra- 
tions. An expensive project is/»not easily replicated. Further, 
an initiative Which is conducted at numerous sites is also dif- 
ficult to replicate because it is difficult to maintain the 
controlled environment necessary for implementation and evalua- 
tion. Moreover, allowing applicants to design their own pro- 
grams without providing any additional guidance would complicate 
evaluation and make it difficult to determine what needs to be 
done to make a project work in other communities. 

RE SEARCH /DEMONSTRATION INITIATIVES HAVE 
NOT BEEN ADEQUATELY DESIGNED AND MANAGED 

Two of the research/demonstration initiatives we examined 
were not adequately designed and managed. Our review surfaced 
five major areas where inadequate design and management hampered 
the achievement of research/demonstration . Specifically: 

— Program announcements failed to clearly establish 
research/demonstration as the initiatives 1 purpose. 

— Research was not adequately incorporated into the ini- 
tiatives' development and design. 

— Evaluation was nop made an integral part of the ini- 
tiatives* 

— Technical assistance wa s not always delivered to 
grantees . 

— The amount and type of monitoring required for 

research/demonstration programs were not determined. 

As a result of inadequate design and management, the initia- 
tives have encountered implementation and evaluation problems 
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that, will lll.it the. achievement o£ tesearch/denonstration 
obj ectives . 

prn gram announceme nts did not 

^alri y establish rese» rrh/demonstration 

as the initiatives' purpose 

According to studies by th« .Band r^ns?"-" 

qu ?^s"'The CT D .^Jo^^" provtoes'anlssess.ent o £ 
current research or a theory base. 

t «Ton rn! 9 tlftir.r:.T.: n ietJ 0 dld tVO sof.fe qU . th te!y rtacllsh 

subsequent solicitations for the «"»l« a "° n plan s and 

as well as OjJoP fiscal 19 , 9 ^ e i?.J 0 f ^tn^e Initiatives 

technical aa.iV«nc« doct»ent«. all i den J^f ^ clotity on 

s! f d £ r.n s t"pu"4..s. p.?,.- !.*-.»»- 



tion . 



^ Tid P enrir y rhrrnirirtlve £0 L M a J re;onsnrtron 0 and in in i ten Ve 
,bjectives. The Prevention initiative s oDjectxve 



objectives. a.— 

in the program announcement were 



•* * * to develop and implement new approaches, tech- 
niques, and methods to prevent juvenile delinquency in 
communities where youths are in 9«Jt danger ° f bec ° rain< > 
delinquent through improving the ability of not tor 
p?ofit youth serving agencies and organizations to 
implement programs which increase or expand social, 
cultural, educational, vocational, r ecreationar-1 , and 
health services for youth." 



1/Walter S. Baer , Leland L. Johnson, Edward W. Marrow, Executive 
- Turn I r v f o ™ An klysis o f Federall y Funded_Demonstr ation Pro- 
ficTpR-l^S-DOC Santa Konica, Ca.: Hand Corporation 
4Ht!'i976> P 7; The Role o f npmonstratlons in Federa l R j_D 
( M oning ton , ~ 57c7~i Of fTce — of _ Technology Assessment , 
. July 1978) , p. 51. 
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The Alter native Education program announcement did id en ti f y 
the initiative as a demonstration, but it, too, had service 
delivery objectives. The program announcement for the Alternaf- 
tive Education initiative stated the effort was x 

j 

■* * * a major demonstration program to prevent juvenile 
delinquency through the development and implementation 
of projects designed to keep students in schools, to 
pr event unwarranted and arbi tr ary suspensions and ex- 
pulsions , dropouts , pu shouts , and truancy . The specific 
objectives are: 

(1) To develop and implement strategies and 
techniques in Al ter native Education in 
public and private not-for-profit schools 
which imprpve those education pol icies , 
practices, and procedures which impact 
services to youth. 

(2) To upgrade the quality of existing alternative 
education by improving cur r icul um development , 
staff training, youth and parent participation, 
and administrative policies and practices of 
school s and school distr icts . 



(3) To reduce the number of student dropouts, 

tr uan ts , suspensions and expulsions in school s 
and school districts where these programs operate, 

(4) To prepare students for employment and/or suc- 
cessful par tic i pa t ion in post- secondary tra in ing 
and education." 



As these statements from the program announcements illustrate, 
the research/demonstration initiatives had multiple objectives, 
but the research/demonstration purpose was not adequately de- 
fined . 

Failure to adequately establish the initiatives' pur- 
pose can lead to implementation problems because the par ties 
responsible for implementing and evaluating the initiative may 
not ag^ee on what was intended. The following problems oc- 
curred dur ing the implementation of the Prevention in i t ia t ive 
which illustrate this point. 

— The evaluator , the technical assistance contractor , 
and the grantees disagreed on whether the initiative's 
pr imary purpose was demons tr at ion or service delivery. 
The evaluator and the technical assistance contractor, 
both viewed the initiative' s pr imary purpose as demon- 
stration while the grantees viewed it as service deli- 
very. The grantees we interv iewed told us that because 
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they understock! the Prevention initiative's primary pur- 
pose to'be service (Jeliver^, they were confused by the* 
evaluator' s attempts to establish research/demonstration 
requirements • A 

— The contractor responsible for providing technical 
assistance to the Prevention initiative noted in a 
draft of its final report that grantees were con- 
fused about OJJDP' s goals for the initiative. The 
contractor notdd that the grantees were confused 
because OJJDP, in emphasizing the number of youths 
served, appeared to support a traditional service 
delivery purpose for the initiative • 

— The ©valuator, the technical assistance contractor, 
and the grantees disagreed on the target grou$? the 
grantees should involve. According to the evaluator, 
the initiative should have involved only juvenile 
delinquency-prone individuals in order to prove that 
the initiative affected delinquency. The technical 
assistance contractor and the grantees, on the other 
hand, told us the initiative should involve all 
juveniles . 

According to the technical assistance contractor, the 
grantees never overcame the confusion regarding the initia- 
tive 1 s purpose. In fact, both .%he eva^ator and the technical 
assistance contractor concludea^in their preliminary reports 
that the grantees generally provided traditional services to 
only a small percentage of juveniles within the initiative's 
target group. 

Confusion also developed over the purpose of the Alter- 
native Education initiative. For example, the evaluator, 
in his analysis of the grantees' proposals, noted that certain 
grantees lacked clear, specific objectives and evaluation 
components. The evaluator told us that several projects' 
objectives were service delivery-oriented and were too nebulcpus 
for evaluation. Officials of one grantee in the Alternative - 
Education initiative told us that OJJDP had asked them to 
accomplish objectives which were not part of their grant pro- 
posal or workplan. They were concerned that their funding would 
be terminated because of objectives they did not adopt or agree 
to fulfill . 

To ensure that parties responsible for implementing and 
evaluating an initiative share the same under stand inq of the 
initiative, the purpose must be clearly established. 




• 
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Research has been misp ing from research/ 
d emonstration in itiatives 

The OJJDP has not sufficiently incorporated research 
into The ajvelo^ent an d design ^J^"* - should 

conducted in the areas °f greatest n d nd J^^tld that a 
developed and designed. 1/ Var lous stu °J e . , throug h re- 
demonstration should ^ started after determining «g 
search, development, and testing that c " e ^ n "°;° Th 

be demonstrated has a high "{•ii^hich started w!tfo>ut 
studies noted that demonst rations wh xch were ■ QJJD p , a 

such a determination ^« "kely to be c p research 

program development and design P r °J^ S r ^ earch/deinons tr ation 

fni t ia£ves^ 

tion objectives. 

' , ^ r-ofUnhP of Juvenile Justice and Delinquency 

emphasis initiatives. The Director of his organization 

tives to only a limited or moderate extent. 

r *u initiatives we reviewed indepth, the 

^wfou.^ireoVp « o nf esearc* We not involved^ 

program decisions which had already oeen m w hich was 

assumptions pertaining to lnltxjti™ be for. th.^ 

"Center S^rbl^SUrU available research ana 



o... Department of Justice >. j|g^T ^^^ ^fto^D .C. GP°. 

^f^nnPop- -74 , HicnaeTKaJ nor , »b.?t Howard 
August 1981 , pp. 7J 741 igniwntatlon Process For a 

ioSSii^^ 
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preparing a background paper for the Alternative Education 
initiative' told ua that their research assessment was not 
conducted specifically for the initiative but rather was 
conducted for an unrelated effort. Further, they told us that 
they were not involved in designing the program requirements 
for the Alternative Education initiative. 

The following are some examples of how limiting research 
in program development and design can cause problems. 

— Two of the three initiatives reviewed indepth were 

implemented as demonstrations without undergoing ^ 
the ^ftkjrst steps in the program development process — [ 
research and testing. In the Prevention initiative, 
improper program development may have caysed the ini- 
tiative to be prematurely implemented. Preliminary 
reports of the evaluator and the technical assistance 
contractor noted that the initiative had prematurely 
promoted delinquency prevention before the concept 
and strategies had been adequately researched and 
understood. Supporting this was the fact that after 

- the demonstration initiative was begun, OJJDP in 1980 
started an initiative entitled "Prevention Research and 
Development" to test specific prevention strategies. In 
the Alternative Education initiative, inadequate program 
development also caused problems with the initiative's 
implementation. For example, the evaluator told Is that 
one grantee was promoting a strategy which research has 
shown to be harmful to juveniles. Further, the technical 
assistance contractor who helped redraft th* guidelines 
told us that the original guidelines had to be redone 
because the OJJDP was relying on antiquated Concepts 
proven ineffective and discarded by the Department of 
Education . 

— 1 In the Prevention initiative, private nonprofit organi- 
zations were required to be used to implement the ini- 
tiative, but those organizations may not have been the 
best suited for conducting research/demonstration. 
Documents we reviewed indicated that this requirement 
was established before the research had been assessed. 
The preliminary reports of the evaluator and the techni- 
cal assistance contractor indicated that large private 
nonprofit organizations may not be the best organiza- 
tions to implement a demonstration since they t<*nd to be 
traditional organizations with little experience assess- 
ing program impact and implementing programs in the poor, 
high crime, inner city areas where youth problems were 
considered most severe. Additionally, the evaluator 
noted that the assertion that private organizations 



were the most capable for carrying out the initiative 
was not based on research. 
-With the Alternative Education initiative ^he bap kg round 
paper noted that most strategies had not been adequ ateiy 
tested and proven effective. Yet, the initiatl ve spec 
aram guidelines required potential grantees to incorpo 
9 ra" 'the strategies 1 discussed in the backg round paper 
Further, the background paper stated that the effective 
«»o nf alternative education programs will remain un 
"known unless "programs incorporate standardized -easures, 
Sgo^ous evaluation design, and adequate followup time 
frames" But, the program guidelines did not require 



them . 



— With the Prevention initiative, we found that OJJDP 
fnowing!y issued a program announcement which was 
considered too vague and contained a PW« 
which was insufficiently rigorous. Dur ing internal 
review the Law Enforcement Assistance Administration s 
Off ice 'for Policy Development criticized the guidelines 
as being loosely drawn, not relating to the act ' s intent , 
Ha hi «ld ol faulty assumptions. The Administrator of 
SSo^SnSlSg^th. guideline's deficiencies b t 
stated that the deficiencies would be cor tec teel later 
since they wanted to commit funds. Our analysis of the 
initiative Indicated this vagueness was never corrected. 
The evLuator for the initiative told us* that the vague 
guidelines encouraged the submission of numerous p o- 
^osals wbic^i were similarly vague and lacking in rigor. 

These examples illustrate how thj . failure to ^JJ^J^ion 
^lative^o be de s ig ned . 

The design of the Violent Juvenile Offender initiative, 
part Twa bleed on "^J^ 

rend! e tha %hT P in n i g tia h t e ive n br f^es^rch' and development effort 
to test specific treatment strategies since "Jtlj Jnowl^dge 

tion model. program w ^ prevent ion com- 

St n a % eg o f S tL er InitiIave; 111 be will integrated with research 
rsTncertafn 6 ^was' still in development at the time of our 



f ieldwor k . 



The OJJDP needs to ensure that its research/demonstration 
initiative^ 1 program development and design are based on ade- 
quate research. This is necessary so that program requirements 
support a research and demonstration purpose- 

Evaluation has not been made %p integral part r ~ 
of research/ demonstration initiatives 

The OJjdt has not made evaluation an integral part of the 
Special Emphasis research/demonstration initiatives. With re- 
search/demonstrations, evaluation is an indispensable aspoqt Of 
the initiative since it is through the evaluation that a Federal 
agency can gain better knowledge, determine the program's effec- 
tiveness, and promote the replication of the programs. A %tudy 
prepared for OJJDP noted that evaluation should be made an inte^ 
gral part of the initiative if OJJDP is to avoid repeating past 
errors. 1/ In the preface to that study, the Administrator of 
OJJDP stated that OJJDP was planning special emphasis programs 
according to the model discussed in the ot^idy and making re- 
search and evaluation an integral part of I'ta special emphasis 
initiatives . ' ^ 

For two of the three initiatives we reviewed, we found that 
the evaluation was not an integral part of the initiative. The 
Prevention initiative was not a model of collaboration between 
the evaluator and OJJDP. Although OJJDP was more supportive 
of the Alternative Education initiative's evaluation, problems 
still existed. Specifically, we found that: 

— The initiatives lacked a standard controlled program 
design which made them difficult to evaluate. 

■ > 

— The OJJDP did not adequately communicate the evaluation 
plan and grantees ' responsibilities for implementing , 
the evaluation . 

— The initiatives lacked grant award selection factors 
which help weed out grantees whose projects could not. 
be evaluated or who are not capable and interested in 
implementing research/ demonstration initiatives . 

— The OJJDP did not involve the ^valuators until after 
the program had already been developed. 



I/Department of Justice Office of, Juvenile Justice and 
Delinquency Prevention, LaMar T. Emn^y , A Model for thq 
Evaluation of Programs in Juvenile .Tuntice , ( Wa sh 1 nqt on , 
0.C-: GPO, January 1°77); p. H 
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As a result, the equators had difficulty [ a f l ^»S."*.SSii 
ble and controlled evaluation of the initiative 
program impact. 

mi nat ives lacked a standa rd 
controlled design 

Establishing a standard controller, program design is 

achieve resear cn/demonstration °^ectives An init _ 
design should be sufficiently standard if the £"9™ 
tiveness. nationwide is to be Jeterminea I th J"^ 1 tha , 
lacks a standard design, OJJDP may ^itti cuicy v ienci 

the initiative is ^VlFSS .fflSiS- for reasons unrelated 
successful results may hav ; n b "" ""^J the resear ch/demonstr a- 
to the program. Further, Jo ensure that the re ' rogram im _ 

(2) sample sizes of suf ^ cien %^"^ U i ev ei s ? ?3) continuance of 
effect with Preestablished confidence ^vels, (3) ior ' ad _ 

the treatmentyfor a suf f ic ijn t amount o ft irte r 

justment, (^observation of behavior over a suff ^ (J) 

to determine^ the ultimate re 5PO ns * or ect ion of data in a way 
administration ot ^ «P« ^f^" Vh^e conditions, though 
that does not contaminate resu ^"- f inirtive data on pro- 

difficult to achieve, provide the mos*t definitive 
gca effectiveness. / 

Both the Prevention and Alternat ive Ed ucat ion init iatives^ 
lacked a standardized progr a, ign. ^noemen?. , allo£ed ? grant 
designs, as contained in tne f«- on4 . -a. ion strateqies which 
applicants to develop specific m f P^ m f n ^ 0n . d ibiy replicated, 
did not ensure that the ^°9 r »" c ?«ul£ they could not provide 
The evaluators told us th£. as a "J*^ each p^- 

i-hP best data on overall impact. Tnis was D =^-°"= 
ject hS a unique design and could not be compared. 

ln - t "5:^iiSt*. "Sent i»p==f iil.t~nn.it.. Both evaluators 
Sj*"£iiSrU e t Svaiuations ^ Jh. action ^ogra-^een^ 
better integrated from the start, tney 
provide-better data 'on program impact. 



v 
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Better communication of evaluation 
requirements is needed 

The OJJDP also needs to better communicate to the grantees 
the evaluation strategy and the grantees 1 responsibilities for 
the evaluation. The program announcements for both the Preven- 
tion and Alternative Education initiatives did not adequately 
describe the evaluation plan and the grantees' responsibilities. 

For example, the program announcements did not inform the gran- 
tees that they would have to administer a questionnaire and 
place the juveniles in control and test groups. OJJDP had prob- 
lems conducting the evaluations. In the reevention initiative 
certain grantees refused to comply with evaluation requirements. 
Further, in the Alternative Education initiative, the evaluation 
questionnaire had to be modified before certain grantees would 
comply. Also, two of the grjantees in the Alternative Education 
initiative told us that they would not have applied for funding 
if they had known what the evaluation would entail. The gran- 
tees we interviewed told us they were not told, specifically 
what the evaluation requirements were until they received their 
grant awards. They believed they should have been informed in 
advance of evaluation requirements. 

Another example of the need for better communication in- 
volves OJJDP' s requirements that grantees set aside a percentage 
of their grant awards for the evaluation. The prdigram announce- 
ments for the Prevention and Alternative Education initiatives 
required that grantees allocate up to 10 percent and 15 percent, 
respectively, of their grant awards for the evaluation. The 
evaluators expected each grantee to set aside the full amount. 
The grantees we interviewed told us they were confused by the 
requirement and thought they were allowed to*»set aside any 
amount with the maximum specified. The OJJDP did not clarify 
the confusion by uniformly enforcing the program guideline's 
requirements. 

Grant award criteria should ipclude a 
factor for evaluation issues 

We found that the grant awar^ selection criteria for the 
Prevention and Alternative Education initiatives lacked a factor 
for determining whether the project /could be evaluated. The 
evaluators for both initiatives told us that, as a result, pro- 
jects selected either could not be evaluate^ or were inappro- 
priate for a controlled evaluation. To successfully implement 
a research/demonstration initiative, the projects selected 
should be capable of being evaluated and, the grantees should be 
capable and experienced. The grant award criteria for initia- 
tives should include a factor that will enable selection of 
capable grantees and grantees with projects which can be , 
evaluated. . ' 
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E valuators were not i nvolved until 
TZT programs had Been developed 

Although th. .v.lu.tor. w.r. "sponsible for d.t.rjlninj 
th. impaot of th. »^iU.^ h "no^nvo!via"n k£ Program 

^.^t t .„ d i s«l^n..:runtix ^tfjssxrj&sr 

developed. Effective integration of the " ° 

evaluation requires that the •'^ c ^° 1 afterwards . 
program phases before the program » ^^^^^^f^ucation 
The evaluators for ftn^d irke? Program and design deci- 
initiatives were not involved in Key p*^*. , 
s?ons 'including defining the ^rget group formulating a 
theoretical statement of the P r ° b i em ' an ^ v V *^ P iS? help to 
strategy and evaluation plan, ^^^^^^^^.termine whether 
review the proposals or visit the grantees to ( des Qr 

the grantees' projects complied with the i^iati j ^ 

would have problems °»tainin|data . have. been minimized if 

problems evaluating the initiatives wouia v review> and 

?hey had been involved in P^ r ™ g^Kd to perform ' 

STr r 4;.^^^ 

to yield the best available information, couj.q « 
or minimized . 

The evaluation aspect was int gJJJ-^^^i-Ttor 
Violent Juvenile Offender, P art * ^i*"^; initiative, eatab- 
helped: define the target P°P ula ^°"fj n ^ ven tion strategy to 
lish the theoretical model, develop t^ ^ventl and 9elec - 
be tested, and establish the ^P^^^P^riteria took into 
tion- criteria. Additionally, Reevaluated. While 

consideration the a PPi ica "t 9 ^itY to the implementation 

part I of the initiative h J ^^Ve" ve that Evaluation 

SSJl^-S-iS:- SiSSS'th. -tion program and the • 
evaluation wezre better linked. 

To ,n.ur.'th.t th. sp.=i.l •^"Vcr^M.'and'convIno- 
an d "Plioation POtsntial i. X»u«l» .'..l.t^.l 

-lard, control^ P™*"™ ? ^"na'tht gr.nt..s ■ r.spon- 
1„ adv.no. on th. .valuation .tr.t.gy include a 

sibiliti.s: r «j; ir >"? " a * h ' r ""o1.c? c« b. .v.lu.t.d: and 
deter rao!no°s.!.otion of c.pabWand .xp.ri.ncd grant.... 
encouraging selection uj- f , M bv devising ways to 

grant'Lard. Tn. .v. 1U atLn 9 should not b. vi.w.d »r.l» as a 
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requirement. Rather, it should be viewed as an integral p-art 
of the program. If OJJDP cannot provfe that the Special Emphasis 
initiatives are worth replicatinq, then its investment' in 
research and demonstration will have a minimal return. 

Technical assistance has not always <^ 
been provided * 

Technical assistance has not always been available to special 
emphasis grantees. Since research/demonstrations are /difficult 
to implement, it is important that technical assistance be avail- 
able. However, in both the Prevention and Alternative Education 
initiatives, we found that grantees did not receive technical 
assistance during key phases of the programs because of problems 
in awarding the contract and in coordinating OJJDP 1 s efforts. 
„ For example: 

> . j 

— The technical assistance contractor for the Prevention 
initiative was responsible for helping to develop the a - 
; strategy for implementing the initiative by providing 
inforxqation on promising delinquency prevention theory ^ 
and ; practice . , The contractor was also responsible 
for helping State and local organizations implement 
sound delinquency prevention programs . To accomplish 
this effectively, the technical assistance cbntractor 
should have been involved early. However, the contractor 
was not on board until the initiative had been imple- 
mented for 1 year. The technical assistance contractor 
told us "that as a result, involvement in strategy develop- 
ment was minimal, 

— With thfi Alternative Education initiative, the Program 
Manager was not apprised of the status of technical assis- 
tance^ efforts . The OJJDP planned to replace the original 
contractor with another firm 7 months after the program 
had been implemented. However, tlie oriqinal contract was 
extended because of problems encountered in awarding a 
contract to the other firm. The Program Manaqer told us 
she was unaware the contract had been extended or else 
she would have; continued referring requests to the oriqi- 
nal contractor/ An official of the contractor told us 
they were able to conduct only an assessment of ^^e^ 
grantees' heeds and* would not have been able to hanole 
additional requests because they w^re winding down their 
activities. Due to the poor coordination between the two 
divisions within OJJDP, the grantees did not receive any 
technical assistance for at least 5 months during the 
program's first yea^ of implementation. 

To ensure that technical assistance is available to qran- J 
tees during all program phases, the OJJDP needs to better 
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coordinate the technical assistance contracts with the special 
emphasis research/demonstration initiatives. 

Amount and type of monitori ng needed . 
for research/demonstratio n initiatives 
should be determined 

As discussed earlier- in this report, OJJDP's policies and 
procedures primarily address a service delivery Program, ^ince 
OJJDP's policies and procedures governing m ° n *t°rinq alsoaddress 
service delivery, a question exists as to whether re " a "^ 
demonstration programs should be monitored in the same manner as 
service delivery programs or given closer scrutiny. nJJDP needs to 
address this issue. 

Also, our review of the Prevention and Alternative Education 
initiatives surfaced basi* problems which need to ^ corrected. 
With the Prevention initiative, no one person w "" fl P 0 "»^ le f ° r 
assessing grantee performance and project compliance. as a 
assessing yi awv.o ^ ouarterlv reports were submitted on 

result, no one made sure that quarterly repv , , - th 

time or used to assess grantee performance. <* r J"* 1 *"* 8 Q ° \™ 
Prevention initiative's quarterly reports shows that out of 144 
reports required during the initiative ' V!f J^l/L aaencv 
and 59 were either never submitted or not doc ^ e "ted i^ agency 
file*. Additionally, officials of one grantee told us they had 

Len a grant monitor from OJJDP and, further, had no idea 
who their grant monitor was. To find out, we contacted th Pro- 
gram Manager of the Prevention initiative and ■• vera i,Spacial 
Sasis' grant monitors, but. nobody knew who was responsible for 
monitoring the grant. 

Our analysis of the monitoring of the Alternative Education 
initiative over the 6-month start up period indicated monitoring 
weakness^. We found that out of 15 site visits required by the 
monitoring plan for the 15 Alternative Education grantees, only 
6 were conducted. The Program Manager told us she could not 
conduct the required site visits due to travel cutbacks. 

Special Emphasis Division officials acknowledged monitoring 
weaknesses but claimed they had been unable t° monitor projects 
adequately because of travel cutbacks and excessive workloads. 
If Sis is the case. OJJDP should explore alternative means of 
keeping track oijgrantees' progress and problems. 

CONCLUSIONS 

Unless OJJDP establishes better policies and Procedures . it 
will achieve only limited results through its special emphasis 
research/demonstration initiatives. We found that due to in- 
adequate policies and procedures, OJJDP has designed and 
managed two of the three research/demonstration initiatives 
ineffectively. Specifically: 
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— '"he program announcements did not adequately establish 
research/demonstration as the initiatives 1 purpose, 
therefore, Xey parties had different views of the 
initiatives 1 purpose . 

— The initiatives' development and design was not 

sufficiently based on research and, as a result, the 
initiatives were not properly developed and designed 
for a research/demonstration . 

— The evaluations might not generate sufficiently credibl 
and definitive data on program impact since the evalu- 
ations were not made an integral part of the initiative 

— The technical assistance to grantees was not available 
since the Special Emphasis Division and the Formula 
Grants and Technical Assistance Division did not ef- 
fectively coordinate technical assistance- 

— The amount and k type of monitoring required for 

research/demonstration programs were not determined. 

The design of the third and most recently developed initiative 
Violent Juvenile Offender, pari: I, was more compatible with re 
search/demonstration conditions which indicates progress is 
being madrf However, OJJDP cannot ensure that progress will 
continue tUgkbe made so long as Its policies and procedures do 
not recognize that research/dertonstration and service delivery 
programs need to be operated differently. 

The need for improved policies and procedures becomes 
more significant in light of the report by the Attorney 
General's Task Force on violent Crime, The report recommends 
that the Attorney General ensure that adequate resources are 
available for research, development, demonstration, and in- 
dependent evaluation of methods to prevent and reduce serious 
crime. Also, by clarifying its policies and procedures, OJJ^p 
will place less of a burden on individual program managers to 
adequately desigft and manage the special emphasis initiatives* 

RECOMMENDATIONS 

We recommend that the Attorney General direct the Adminis 
trator of OJJDP to establish policies and procedures which 
recognize that research/demonstration and service delivery 
programs should be designed and managed differently- With 
respect to OJJDP 1 s research/demonstration initiatives, the 
Administrator of OJJDP shoiald ensure that: 

— Program announcements c 1 early establish the initiatives 
purpose . 
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-Research/demonstration initiatives are developed on the 
basis of research results. 

-Evaluation is made an integral part of research/ 
demonstration initiatives. 

-Technical assistance is available to grantees 
especially during key phases. 

The Administrator should also ^^^^"^^^^^"rservice 
( demonstration programs should be monitored the same 
^ delivery programs or given closer scrutiny. 

AGENCY COMMENTS AND OUR EVALUATION 

.he Department of Justice commented on ^"ft^f ££ rtlMBt 
report by letter dated March 3, 1982. (See app , lly foCU8e <i 

ag?eed that OJJDP's policies and P r °cf u "shave palmar Y 
upon development and implement ation °< »«^% 0 ^pLated dif- 

S5^^i5»ri^U4Si p SS...i. pro*- 

Mthouch the 0ep.rtn.nj »9".d with J«.~«i»^; 

n . 8d tor pond.. «- "^S";:^ his £<* b .« 

tion that rtwarch « nd ^"™° ns "oSnent. stated that this 
qu.teiy designed and ™n.^ ; The lnltlativ „, but tn . t 

conclusion wa. baaed on "Views Education— were not in- 

With respect to the Prevention Si^^^ISS^ 
justified its Po-^ion ^^^J^^^L intended to "increase 
ment clearly -aid ^^j/^c.Uon.?. vocational, recre- 
or expand social, cultural, i vout h " 'Hie Department stated 
ational, and health £ t £n Sis n ot mentioned in the 

further that research/demonstration was no However , the 

announcement because that was not its P^ S / ann ^ cement wan " 

believe "at this part of the objective connotes research/ 
demonstration. 

*. Alternative Education initiative, the 

S^j^Sip-^ tjro-jJ - % -: n - n -r?Lted j :nr 

deoigned to keep students in school, to prevent » and 
arbitrary suspensions and expulsions, dropouts, p 
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truancy." Also, as pointed out on page 11 of our reoort, 
solicitations for the evaluation of both initiatives, as well 
as OJJDP's program plans and technical assistance documents 
identified the initiatives as demonstrations. 

We continue to believe that our position that the two 
initiatives were intended to be research/demonstration-oriented 
was well-founded. But, regardless of what the initiatives 
were supposed to be, it is clear that there was not a common 
understanding among all participants, of the purposes to be 
achieved. The corrective action promised by OJJDP should help 
to avoid confusion in the future regarding the purpose of the 
various special emphasis initiatives that are funded under this 
program. 

The Department stated that facts reqa^ding the role of the 
National Institute of Juvenile Justice and Delinquency Preven- 
tion were inaccurate. It noted that the Institute had assigned 
a staff person to share responsibility for development of each 
special emphasis initiative* We do not disagree with the 
Department that astaff member was assigned to do thLs. How- 
ever, as noted on page 14 , 'the Director of the Institute told 
us th&t the, Special Emphasis Division had involved the Insti- 
tute to only a limited or moderate extent in the development 
and design of most special emphasis initiatives. 

The Department concurred that , research/ demonstration pro- 
grams should flow from research on a major issue, evaluation 
should be an integral part of these initiatives, technical assis- 
tance should be provided grantees, especially at key phases, and 
monitoring should be intensive. It also offered several comments 
to explain its actions or positions regarding evaluation, techni- 
cal assistance, and monitoring issues discussed in our report, 
* 

The Department also noted that our recommendations appeared 
to lump all the initiatives together leaving the impression that 
the Violent Juvenile Offender initiative did, not adeauately 
establish its purpose in the request for proposals, was not based 
on research results, and did not make evaluation an inteqral 
part of the initiative. ^Tiat was not our intent. We had notei 
in the draft report submitted to the Department for comment that 
the design of the Violent Juvenile Offender, part I initiative 
was compatible with research/demonstration conditions. We pointed 
out that to help ensure that future reoearch/demons tration ini- 
tiatives will be desiqned to comply with such conditions, OJJnp's 
policies and procedures should recognize that research/denon- 
stration and service delivery programs need to be operated 
differently. 
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3) Congress and the public expect that funds allocated in a given fiscal year be obligated 
in that fiscal year. The option of holding funds until the moot reliable data become 
available is not a viable alternative. 

Design and Management of Research/Demonstration Programs 

We have difficulty with 0X0*3 finding that research and demonstration programs hive not 
bean adequately designed and managed and do not feel that the information used supports 
this conclusion. GAO*s conclusions are based upon In-depth reviews of three initiatives, 
only one of which was intended to be a research and development program, I.e., Violent 
Offender Part I and Part EL 

1. Prevention* Youth Skills Development 

Contrary to the QAO report, the program announcement for the Prevention- Youth 
Skills Development initiative clearly states In the program objective (page 59 of the 
announcement) that the program was Intended to "increase or expand social, cultural, 
educational, vocational, recreational and health services to youth." This program was 
specifically designed to implement Section 224 (a) (4) of the Juvenile Justice and 
Delinquency Prevention Act, which requires improving the capability of public and private 
agencies and organizations to provide services for delinquents and other youth to help 
prevent delinquency. Research/demonstration was not mentioned in the announcement 
because that wo not its purpose, and statutory provisions clearly support OJJDP funding 
service delivery programs. 

State of the art information provided the basis for the program's strategies, but no clear 
prevention models were available from research in 1979 for testing. Evaluation was 
provided for in the program announcement, but was not made an integral part of the 
initiative because the evaluation grant had not been awarded when the program was 
announced. A level of monitoring had been established consistent with the funds 
expended, but other administrative priorities intervened which did not permit the planned 
monitoring to take place. As a result of staff shortages and turnover, several prevention 
grants were subject to changes in monitoring assignments, but no grant was left without 
an assigned monitor. Technical assistance is voluntary, and was available upon request 
after the technical assistance contract was awarded. Some grantees made more use of 
this assistance than others. 

2. The Alternative Education Program 

The Alternative Education Program announcement states in the objective state- 
ment that it is a major demonstration program to prevent delinquency "by preventing 
unwarranted and arbitrary suspensions and expulsions, dropouts, push outs and truancy. n 
Unlike the Violent Offender Program, it was not intended to be a research program, but 
one which would implement a number of promising strategies in varied school districts. 
The evaluation design was not an integral part of the program design because the 
evaluation grant had not been awarded when the program was announced. However, the 
goals of evaluation were stated, and grantees were required to document support for 
evaluation objectives. As with The prevention program, research had not validated any 
single model of alternative education when this program was announced, but available 
state of the art information was used as the basis for the program's strategies. With 
respect to level of monitoring, this program is subject to an extensive program 
development process funded through the evaluation grant, which was expressly developed 
to support strengthening of the objectives. It has, as an integral part, an extensive 
management information system. This system supports staff monitoring, and OJJDP feels 
that the level of monitoring currently taking place is adequate. 
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3. Violent Juvenile Offender Part I and n 

~\ Violent Juvenile Offender Part I and 0 is a research and development (R&D) 
program, not research/demonstration as categorized in the G AO -report* Its objective is 
to jest program models based upon selected theoretical constructs. If research validates 
thdt the models beir^ tested have merit, future demonstration programs may be 
-appropriate. Procedures for carrying out the work are outlined in the contracts awarded 
for the developmental and implementation work to the National 'Center for Crime and 
Delinquency (NCCD) and L. Miranda, Associates. Both contracts provide for a high level 
of monitoring, technical assistance, and administrative support throughout the imple- 
mentation phase. 

Rec om m endat ions 

As a matter of practice, over the past two years, OJJDP has moved toward implementing 
the recommendations of the OAO report, with increased priority given to testing program 
models based upon research findings. Violent Juvenile Offender Part I and Part n, 
developed In 1981 for implementation in 1982, the Prevention Research and Development 
Program, developed in 1980, and the Replication of Project New Pride, an exemplary 
program model being replicated in seven sites, are examples of this focus. 

While OAO recognizes that the Violent Juvenile Offender Program was developed as an 
R&D program and that the evaluation and research requirements are stated in the background 
paper, the report does not point out that the RAD requirement was clearly spelled out in 
the Request For Proposals (RFP) which was jointly developed by OJJDP; National Council 
on Crime and Delinquency, program coordinator; URSA Institute, the national ©valuator; 
and the National Office for Social Responsibility, the technical assistance contractor. 

The reports recommendation section appears to lump all the initiatives together, leaving 
the impreaeion that Violent Juvenile Offender Part I (!) did not adequately establish Its 
R<3cD purpose in the RFP, (2) was not based on research results, and (3) did not make the 
evaluation an integral part of the R&D initiative. None of these conclusions apply to the 
Violent Juvenile Offender initiative. 

OJJDP will formalize policy and procedures for development and implementation of the 
several categories of programming possible under Section 224 (a) within the constraints of 
increasingly limited staff resources and reduced travel funds, both of which will continue 
to impose constraints on the level of monitoring possible. Responsibility for establishing 
precise procedures for monitoring has been placed with contractors to help fulfill OJJDP 
monitoring responsibilities. NCCD has established such procedures already and the 
contractor for Part fl is expected to do the same. Finally, OJJDP wishes to note that 
OJJDP* priorities, as well as the expectations of Congress and the general public will 
continue to influence the priorities for use of funds available for special emphasis 
programs, and that our policies and procedures will of necessity be responsive to these 
cone cms v 

The Attorney General's Task Force on Violent Crime has recommended that R&D be 
viewed as an appropriate Federal function, and OJJDP will intensify its efforts to be more 
responsive to this viriw. Further, the availability of more reliable data now makes it 
possible to base demonstration programs upon research findings, and reduced funding 
makes it necessary to narrow the focus and scope of special emphasis programs. 
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COMMENTS BY ISSUE * 

1. OJJDP^ Policies and Procedures Do Not Recognize the Need to Operate Research / 
Demonstration and Service Delivery Programs uiffcrently , 

OJJDP concurs that the procedures and policies provided in our Special Emphasis 
Operations Manual focus largely upon service delivery programs. Violent Juvenile 
Offender, one of three research /demonstration programs funded by OJJDP in the 
past seven year^the others were Prevention Research/Development and Learning 
Disabilities— has as part of the cooperative agree mentMth NCCD, specifications 
for development and management of Part I of this program. These specifications 
include all aspects of development, evaluation, monitoring and technical assistance. 
Hie contract with L. Miranda, Associates has similar provisions for Part TL 

The thrust of OJJDP 1 * special emphasis programming has focused on system change 
and support of national policy as reflected in the Juvenile Justice and Delinquency 
Prevention Act. Accordingly, most of Its resources have supported alternatives to 
incarceration (Deinstitutionalization of Status Offenders, Diversion, Restitution, 
New Pride, and Jail Removal). The strategy for doing this is reflected in those 
elements which are identified as not t>eing suitable for research and demonstration 
programs, La., targeting specific categories of youth, sizeable grants, multiple sites, 
multiple year funding, and program designs responsive to local conditions. The 
latter provision recognised that no single model would work in every jurisdiction, 
and that local support and participation in programs is directly related to the extent 
to which local residents feel that the program responds to conditions peculiar to 
their Jurisdiction. 

While program models have varied, OJJDP regards the enactment of state statutes 
and codes to allow for alternatives to incarceration, like restitution and diversion, 
as significant. The significance for national policy and large numbers of youth takes 
on added importance when these state statutory changes have been accompanied by 
allocation of state funds to support these alternatives. 

2. Research/Demonstration Initiatives Have Not Been Adequately Designed and 
Managed 

We take exception to the conclusion that research/demonstration initiatives have 
not been adequately designed and managed, as only one of the three programs 
examined in depth is classified by OJJDP as a research/development program, I.e., 
Violent Juvenile Offender Part I and Part IL The other two programs which were so 
classified, Learning Disabilities and Prevention Research/Development, were not 
examined. Therefore, the finding that the program announcements failed to 
establish research/demonstration as the purpose, is inappropriate to Prevention- 
Youth Skills and Alternative Education. Conversely, these were not conclusions 
made about the Violent Juvenile Offender initiative, which Is a research program. 

3. Program Announcements Did Not Clearly Establish Research/Demonstration as the 
- Initiatives' Purpose 

As stated above, research was not the purpose of either Prevention-Youth Skills or 
Alternative Education. Therefore, it would have been inappropriate to have 
included references to research/demonstration in the program announcement. 
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Prevention- Youth Skills was conceived and designed as a service delivery program to 
implement Section 224 (a) (4), which requires improving the "capability of public and 
private agencies and organizations to provide services for delinquents and other 
youth to help prevent delinquency"; and Section 224, (a) (c), which in 1977 required 
"at least 25 per centum of funds available for grants and contracts ... to institutions 
who have had experience in dealing with youth." A reference on page 14 of the 
report, which questions the appropriateness of national youth-serving agencies 
implementing this program, discounts the fact that this section of the legislation, 
and these provisions specifically, were included for purposes of supporting national 
youth-serving agencies in delinquency prevention activities. 

Contrary to the view of the national evaluator, NCCD, the initiative was not 
designed to test a theory of delinquency prevention. In retrospect, the initiative 
was not ideally suited to a national scope evaluation, and might more profitably 
have been restricted to project level evaluation. The program office monitored 
projects in relation to accountability for service delivery, a grant condition, and the 
purpose of the program* 

Alternative Education was not intended to be a research program. Rather, it is a 
service delivery /demonstration program which states, in the program announcement, 
as specific objectives: Vs 



1) "To develop and implement strategies and techniques in Alternative Education . . . 
which improve those educational policies, practices and procedures which impact 
services to youth." 

2) "To upgrade the quality of alternative education program . . . ." 

3) "To reduce the number of student dropouts, truants, suspensions and 
expulsions . . . ." 

4) "To prepare students for employment 

Adequate research findings were not available when the program was developed to 
support a research/demonstration program. The program design drew upon current 
state of the art information available to N1JJDP. The program was developed in 
response to Section 224 (a) (6) which requires that n . . . model programs and methods 
be developed to keep students in elementary and secondary schools." Consequently, 
OJJDP was interested in evaluating a number of alternative education models, as 
opposed to a single model, further reflecting the wide variation in the nation* 
school districts. 0 



OJJDP agrees that the objectives of some alternative education projects lacked 
specificity. One of the specific tasks of Johns Hopkins was to work with each 
project to make objectives more specific. Funds were made available to grantees 
for this purpose. 

4. Research Has Been Missing Prom Research/Demonstration Initiatives 

Research has been incorporated in each initiative to the extent that valid research 
findings were available. Facts regarding the roles of N1JJDP and the American 
Institute for Research (AIR) in the development of special emphasis initiatives are 




inaccurate. 
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1) MIJJDP has assigned a staff person to share responsibility for development of 
each special emphasis Initiative, This person is a working member of a task 
group, snaring full responsibility for development of the program announcement. 
HUJDP has also contracted for a background paper for each Initiative, and the 
assigned naif person has responsibility for assuring that this Information is 
available to support the design of the program. 

i 

2) AIR was contracted to provide state of the art work in six program areas, as a 
preliminary ttep to completing the 1977 program plan. A change in administra- 
tor! resulted In shelving this work, and work on alternative education was not 
resumed until late in 1979, long after the AIR contract had terminated. 

Evaluation 4 * 

Contrary to OAOI finding that evaluation is not an integral part of Alternative 
Equation, the evaluation grant awarded to Johns Hopkins University, with a sub- 
contract to the Social Action H es s arch Center (3 ARC), provides for an extensive 
management Information system (MIS), regular on-site vfcits by evakiatlon staff, and 
an extensive program development process built upon MB data. While the program 
design varies In response to the peculiarities of local school conditions, Johns 
Hopkins has a solid evaluation design which tracks similarities, as well as differ* 
enoee, serosa program*. OJJDP feels that the results will provide ueeful data on 
what kind of alternative school models work in relation to characteristics of varied 
education systems. We would, however, agree that it would have been useful to 
had the evaluation grant in place before the announcement wes developed 

Technical Assistance 

111 1 ■ 

to both Alternative Education and Prevention, technical aasta£os was/Is available, 
but was/to voluntary. Some grantees make more extensive uee^f ft than others. 

Monitoring 

One site visit per program year, per project was planned for Prevention, but other 
administrative priorities Intervened and the schedule wes not possible. Staff 
shortages and staff turnover resulted In the assignment of several different grant 
monitors* but at no time wes any Prevention grant without a grant monitor. 

Similarly, one visit per school year was planned for Alternative Education, and thus 
far, OJJDP is on schecfcilo with this plan. Staff monitoring is supported by monthly 
MB reports, and more frequent site visits by the evaluation staff. Moreover, the 
•valuators conAict weekly telephone interviews with grantees to update the MIS 
data, and provide program progress reports to Improve onsite program management. 
OJJDP believes that the level of monitoring is adequate on Alternative Education 
and, moreover, Is all that our travel budget will allow. 

Recommendations 

OJJ DP con curs with all of QAOM recommendations, La., that researeh/demomtra- 
tion programs should flow from research on a major Issue, evaluation should be an 
Integral part of these intietives, there should be a controlled design, and the rrant 
award should include the program's evaluability. We also agree that technical 
assistance should be provided grantees, especially at key phases, and that monitoring 
should be intensive. 



/ 



*GAO note: The report han boon clarified with ronooct to thin romnont 
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These are conditions which haye been, met in the Violent Juvenile Offender 
initiative, and the priocedures* for these conditions are explicit in the contractual 
language of the cooperative agreement with NCCD and Lr. Miranda, Associates. . 

To avoid confusion iri the future regarding the several categories of programming 
possible under special emphasis programs, OJJDP will develop policy and procedures 
regarding purposes and standards for each type of programming- 



Should you desire any additionakinformation pertaining to our response, please feel free 
to contact me* 

Sincerely, 



Kevin D. Rooney vl 
Assistant Attorney General 
for Administration 
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